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Study Protocols 

The following sections provide the protocols taken while conducting evaluations and testing of the Town 
Crier application. Note that the retrospective testing, think-aloud evaluation, and usability testing 
sessions made use of a written script, varying only slightly between each test. Materials used for these 
evaluations can be found at the end of this document in the "Materials" section. 

Heuristic Evaluation 

A heuristic evaluation was conducted with a group of usability experts. Group 2, "The Advisors", agreed 
to conduct a heuristic evaluation on the Town Crier application. The evaluation was conducted in a 
sterile environment, found in the Boyd 307 lab, and lasted for approximately one hour. They were given 
the following task list to guide their interaction with the Town Crier application: 

• The first task is to create a profile for the account. 

• Now that you have created a profile you should see the home page. I would like you to take a 
few moments to familiarize yourself with the page. 

• Please find the restaurant Mama's Boy. 

• Now could you please search for all secret spots that have the tag "Outdoors?" 

• Now could you please find a friend in your list of friends? What restaurants/venues have they 
visited recently? 

• Now please find a different secret spot and its description. 

• Now could you please write a review of the secret spot you found in the previous task? The 
content of your review does not matter we are only interested in the process of posting the 
review. The review will be deleted after this session. 

• Now please think of your favorite park and add it as an attraction. Again, the place itself does 
not matter we are only interested in the process of adding an attraction. The attraction will be 
deleted after this session. 

Figure 1. Heuristic Evaluation Template 





Related 


Priority of 


Description of the problem 


Heuristic Violated 


Potential solutions to the problem 




Problems 


the problem 


A summary of the problem 


The name of the heuristic 


Hypothesize up to 3 solutions to the 




List the #s 


1-4, where 




that this problem violates 


problem 




of any 


1= highest 










problems 


priority (huge 










that are 


flaw!) 










related 










e 

X 

a 










Change the icons. One option is to 


m 






The same icon is used for 




change the "increase text size" to a 


P 
1 






"increase text size" and "select 
text color" (both use a large letter 




small and large A. Another option could 
be to change "select text color" to three 


e 


none 


3 


A) 


Consistency and standards 


A's of various colors. 



Page I 3 



As the group of experts was given the task list, they were also presented with the Town Crier application 
on a conductor's device. They were then told to begin the heuristic evaluation while following the 
guided tasks above. A template was provided to assist in writing down usability problems and elements 
that violated heuristic principles. 

A minimum of one hour was allotted to the group of experts to interact with the application, find 
usability problems, and assign each problem a severity rating. Shortly after an hour, the group of experts 
notified us that the application's evaluation was complete and the template had been successfully filled 
in. They were thanked for their time and compensation was provided in the form of acting as experts for 
their heuristic evaluation. 

Predictive Evaluation - KSLM 

The estimated time each task can be completed by an expert user is found below. Each task uses time 
calculated from the following table. A comparison of the KSLM evaluation and user performance during 
the usability testing session can be found in Figure 4 in the "Discussion" section of this document. 

Figure 2. Time elapsed for each sub-taslc. 
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Tos/f A - Create Profile. 

1. MP (point to Create profile) 

2. PI (click Create Profile) 

3. MP (point to the 'name' field) 

4. PI (click on the 'name' field) 

5. H (hand from mouse to keyboard) 

6. MK(ll) (type in name) ("Kyle Sweeney") 

7. H (hand from keyboard to mouse) 

8. MP (point to the 'email' field) 

9. PI (click on the 'email' field) 

10. H (hand from mouse to keyboard) 

11. MK(ll) (type in email) ("kds@uga.edu") 

12. H (hand from keyboard to mouse) 

13. MP (point to the 'password' field) 

14. PI (click on the 'password' field) 



15. H (hand from mouse to keyboard) 
15. MK(8) (type in password) ("********") 

17. H (hand from keyboard to mouse) 

18. MP (point to the 'interests' field) 

19. PI (click on the 'interests' field) 

20. H (hand from mouse to keyboard) 

21. MK(12) (type in interest) ("hiking, dogs") 

22. H (hand from keyboard to mouse) 

23. MP (point to the 'friends' field) 

24. PI (click on the 'friends' field) 

25. H (hand from mouse to keyboard) 

26. MK(15) (type in friends) ("Hanz Deinfreund") 

27. H (hand from keyboard to mouse) 

28. MP (point to 'add' button) 

29. PI (click 'add' button) 

12M, 7P, 7P1, lOH, 5K(n) 

12(1.35) + 7(0.40) + 7(0.20) + 10(0.40) + 
(57)(0.22) 

16.2 + 2.8 + 1.4 + 4 + 12.54 = 36.94 seconds 

Task B - Find the restaurant. Mama's Boy. 

1. MP (point to 'Categories') 

2. PI (click 'Categories') 

3. MP (point to 'Dining') 

4. PI (click 'dining') 

5. MP (point to 'Mama's Boy') 

6. PI (click 'Mama's Boy') 

3M, 3P, 3P1 

3(1.35) + 3(0.40) + 3(0.20) 
4.05 + 1.2 + 0.6 = 5.85 seconds 

Task C- Search for all spots with the tag "Outdoors." 

1. MP (point to 'Categories') 

2. PI (click 'Categories') 

3. MP (point to 'Secret Spots') 

4. PI (click 'Secret Spots') 

5. MP (point to 'Filter by Tag') 

6. PI (click the 'Filter by Tag' field) 

7. MP (point to 'Outdoors') 

8. PI (click 'Outdoors') 



4M,4P, 4P1 
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4(1.35) + 4(0.40) + 4(0.20) 

5.4 + 1.6 + 0.8 = 7.8 seconds 

Task D - Find a friend on friends list. 

1. MP (point to 'Profile') 

2. PI (clicl< 'Profile') 

3. MP (point to a friend) 

4. PI (click on a friend) 

2M, 2P, 2P1 

2(1.35) + 2(0.40) + 2(0.20) 
2.7 + 0.8 + 0.4 = 3.9 seconds 

Tos/f E - Find a different secret spot. 

1. MP (point to 'Categories') 

2. PI (click on 'Categories') 

3. MP (point to 'Secret Spots') 

4. PI (click on 'Secret Spots') 

5. MP (point to 'Ben Burton Rope Swing') 

6. PI (click on 'Ben Burton Rope Swing') 

3M, 3P, 3P1 

3(1.35) + 3(0.40) + 3(0.20) 

4.05 + 1.2 + 0.6 = 5.85 seconds 

fos/f f - Write a review of ttie secret spot found in Tasl< E. 

1. MP (point to 'Add comment' field) 

2. PI (click on 'Add comment' field) 

3. H (hand from mouse to keyboard) 

4. MK(9) (type in comment) ("I had fun") 

5. H (hand from keyboard to mouse) 

6. MP (point to 'Add' button) 

7. PI (click 'Add' button) 

3M, 2P, 2P1, 2H, K(9) 

3(1.35) + 2(0.40) + 2(0.20) + 2(0.40) + 9(0.22) 
4.05 + 0.8 + 0.4 + 0.8 + 1.98 = 8.03 seconds 

Tasl< G - Add a secret spot. 

1. MP (point to plus sign icon in top right corner) 

2. PI (click plus sign icon) 

3. MP (point to 'name' field) 

4. PI (click on 'name' field) 

5. H (hand from mouse to keyboard) 

6. MK(8) (type in name) ("My House") 
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7. H (hand from keyboard to mouse) 

8. MP (point to 'tag' field) 

9. Pl(clicl<on 'tag' field) 

10. H (hand from mouse to keyboard) 

11. MK(9) (type in tags) ("cool, fun") 

12. H (hand from keyboard to mouse) 

13. MP (point to 'opens' field) 

14. PI (click on 'opens' field) 

15. H (hand from mouse to keyboard) 

16. MK(1) (type in opening time) (8) 

17. H (hand from keyboard to mouse) 

18. MP (point to 'closes' field) 

19. PI (click on 'closes' field) 

20. H (hand from mouse to keyboard) 

21. MK(1) (type in closing time) 

22. H (hand from keyboard to mouse) 

23. MP (point to 'description' field) 

24. PI (click on 'description' field) 

25. H (hand from mouse to keyboard) 

26. MK(29) (type in description) ("My house. It's a cool place!") 

27. H (hand from keyboard to mouse) 

28. MP (point to 'Add' button) 

29. PI (click 'Add' button) 

12M, 7P, 7P1, lOH, K(8+9+l+l+29) 

12(1.35) + 7(0.40) + 7(0.20) + 10(0.40) + 48(0.22) 
16.2 + 2.8 + 1.4 + 4 + 10.56 = 35.96 seconds 

Retrospective Testing Interview 

The retrospective testing interview was conducted in a closed environment - a study room in the SLC. 
Potential subjects were asked to participate in a study that would benefit the design of an application 
focused on venues in the local Athens area. Participants that agreed to the study would be then notified 
of the manner of the testing, involving using the application while being recorded and then watching the 
recording during a post-interaction interview. If the participants agreed to still conduct the study, they 
were then given a consent form to sign, asking their permission to be recorded. The consent form can be 
found in the "Materials" section of this document. 

After the participant has signed the consent form, they were then seated in front of the device used to 
demonstrate the Town Crier application. At this time, the conductors of the test began to use a script to 
formally introduce the user to the study as well as outline the tasks to be completed. The script can also 
be found in the "Materials" section of this document. Video recording also began at this point - these 
recordings include screen capture of the user's interaction with the application as well as video capture 
of the participant (via webcam). 
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A summary of the tasks the participants completed are shown below: 

• Create a profile. 

• Find "Mama's Boy." 

• Find all the secret spots tagged with outdoors. 

• Find a location that a friend of theirs had been at recently 

• Add a comment to a secret spot 

• Add a secret spot. 

Participants were allowed to ask questions while completing the tasks. Once the subject has completed 
the list of tasks, the screen capture of their interaction with the application was stopped. The participant 
was then notified that the post-interaction interview with a conductor would begin. One conductor and 
the participant then replayed the usage of the application - the other conductor would step aside and 
simply take notes. 

During the playback of the video, the participant was asked to explain the choices they made while 
interacting with the application. Questions varied on the participant's responses and interaction, but 
included: 

• "Why did you choose that option to find a restaurant?" 

• "What was the difficulty in adding a secret spot?" 

• "Is this option found on a page you would expect to find it?" 

The participant's responses to these questions were recorded by the conductor not interacting with the 
participant. All responses and interactions were also recorded video capture (via webcam). Once the 
interview was completed, the conductors gave participants a chance to ask any questions before the 
study was terminated. After all questions were answered, the participants were asked to complete a 
short questionnaire about the Town Crier application. The document was created as a Google Form and 
was presented on a conductor's laptop. This document can be found in the "Materials" section. Once 
complete, the participant was thanked for their time and was notified that the retrospective testing 
interview was complete. 

Think-aloud Evaluation 

The think-aloud evaluation was conducted with five participants in an isolated apartment room. Possible 
subjects were asked to conduct a think-aloud evaluation of the Town Crier application, which consisted 
of completing a set list of tasks while verbalizing any comments, thoughts, or complaints. These 
participants were also notified that their comments would be captured on a video recording of their 
interaction with the application and conductor of the test. The chosen five participants were then given 
a consent form to sign and were encouraged, again, to verbalize their thoughts while interacting with 
the application. Once the consent form was signed, the evaluation began. 

The conductor of the evaluation followed a written script and list of tasks. These materials can be found 
in the "Materials" section of this document. A summary of the tasks are included below: 

• Create a profile. 

• Find "Mama's Boy." 

• Find all the secret spots tagged with outdoors. 

• Find a location that a friend of theirs had been at recently 
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• Add a comment to a secret spot 

• Add a secret spot. 

The participant was then seated in front of the device with the Town Crier application and was asl<ed to 
wait for the conductor to begin the evaluation. A script was followed, along with a task list, to guide the 
participant through the interaction. This document can be found in the "Materials" section. The 
participant was encouraged before every task to speak out loud and if there was silence, would be asked 
the following questions: 

• "Do you have any comments on how you completed this task?" 

• "Did you have any difficulty in completing this task?" 

• "Would you do anything differently when trying to complete this task?" 

All comments and answers to questions were written on paper by the conductor. Responses were also 
captured by the video recording and was later used to clarify the notes taken during the interaction. 
Once the list of tasks was completed, the participants were given one last chance to voice any 
comments, thoughts, or ask any questions before the evaluation was terminated. Again, these questions 
were noted and answered by the conductor. When the evaluation was terminated, each participant was 
asked to complete a short questionnaire about their interaction with the Town Crier application. Once 
complete, the participants were thanked for their time and were notified that the think-aloud 
evaluation was complete. 

Usability Testing Session 

Similar to the previous, think-aloud evaluation, the usability testing session was conducted in an isolated 
apartment room with five chosen participants. Possible subjects were asked to participate in a testing 
session for the Town Crier application, which consisted of completing a list of tasks while being video 
recorded. Participants were notified that their performance while interacting with the application would 
be recorded by a conductor and be captured on video. They were also notified that certain tasks would 
be done on a third party application. Yelp, in addition to the Town Crier application. The chosen five 
participants were then given a consent form to sign, asking for their permission to be recorded. Once 
signed, the participant was seated in front on table that had the Town Crier application on one device 
and the Yelp application on another device. 

The session was guided by a task list and a script found in the "Materials" section of this document. As 
the user completed tasks, three performance measures were being recorded on paper by the 
conductor; the time between a user starting and completing a task, the number of wrong pages the user 
visited to complete a task, and the total number of pages visited while completing a task. All three 
performance measures were captured for both the Town Crier application and for certain tasks on the 
Yelp application. The conductor of the usability testing session used a stopwatch to measure time 
elapsed. To record number of pages visited, the conductor observed the participant's interaction with 
the application and used the recording for any data clarification needed. 

Each participant had a different pace while completing each task - some took their time to explore the 
application and enter text, some went through the application very quickly. As such, measuring only 
time elapsed would provide inaccurate results. Therefore, number of pages visited was also recorded to 
have another form of comparison between each task. 
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The performance measure of each task was recorded in the following format: 



Task 


Time 


Error Pages 


Total Pages 


A - Town Crier 


0.00 seconds 


0 errors 


0 pages 


A -Yelp 


0.00 seconds 


0 errors 


0 pages 



A summary of the tasks is shown below: 

Task A - Create a profile. 
Task B - Find "Mama's Boy." 

Task C - Find all the secret spots tagged with outdoors. 
Task D - Find a location that a friend of theirs had been at recently 
Task E - Add a comment to a secret spot 
Task F - Add a secret spot. 

To complete a task, the participant would first use the Town Crier application. The performance 
measures would then be recorded by the conductor and would ask the participant to do the same task 
on the third party application, Yelp. Again, the performance measures would be recorded by the 
conductor. For tasks A through C, Yelp was part of the interaction. For tasks D, E, and F, only the Town 
Crier application was used due to the limitations of Yelp. 

Once all tasks were complete, the participant was given the chance to ask any questions concerning 
their interaction with the application and performance measured during the session. Once these 
questions were answered, the participant was notified that the usability testing was complete. They 
were then provided a short questionnaire to complete concerning their interaction with the Town Crier 
application. After completing the questionnaire, the participant was thanked for their time and was 
notified that all activities for the session have been completed. 

Demographics 

Participants of the retrospective testing interview, think-aloud evaluation, and usability testing session 
consisted of fellow students that agreed to be part of the studies. The age of these participants ranged 
from 18 (freshmen) to 23 (senior). These participants varied in educational background - of the 
participants that disclosed their major, answers included Computer Science, Cognitive Science, 
Environmental Engineering, Civil Engineering, and Public Relations. All participants disclosed that they 
frequent local venues in Athens at least once a week, with the max number of visits exceeding seven 
times a week. 

The group of experts that conducted the heuristic evaluation consisted of three members of Group 2, 
"The Advisers." All three students had Computer Science backgrounds and were well versed in the 
usability principles as outlined by Jakob Nielsen, co-founder of the Nielsen-Norman Group and creator of 
the Heuristic Evaluation. 
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Task Justification 

Since the Town Crier application has features that are already available in frequently used applications, 
there were common tasks that the Town Crier application needed to accomplish in order to compete. 
With this in mind, we asked the users to do each task on our application and one of its competitors. 
Yelp. The first task was creating a simple profile. This task has to be done when the user first uses the 
application. The application has features that allow you to see what your friends are doing and what 
reviews of attractions they have written. So by clicking the "Create Profile" button the user sets up his 
home page which will then display some of the latest happening of friends. This is similar to what is 
required to view friends by the applications competitor. Therefore, this helped display how we 
accommodated a similar feature. The user can also select to login with already existing social media 
accounts from Facebook and Twitter. 

The second task was to find and select a certain restaurant. The user selects the "Categories" button and 
then selects "Restaurants". This will bring up a number of local restaurants to view. Selecting the 
indicated restaurant brings up its information page. The information page gives all the pertinent details 
about that attraction. Here we show that the Town Crier application can accomplish similar simple tasks 
to its competitors in addition to a feature specific to the Town Crier application. 

After the second task, the participant is tasked to do something that we do that the competitor. Yelp, 
does not offer. The third task was to search for a "Secret Spot" by one of the tags used to describe the 
attraction when it was created by a user. We asked them to search for secret spots that were tagged as 
"Outdoors". This will bring up a collection of secret spots that can be created by users. This feature sets 
us apart from our competitor because secret spots are not a category of which other competitors have. 
This category is specifically for local venues in Athens that not many know about. In addition, these 
spots do not need to be businesses. It can be odd locations in Athens, such as the Iron Horse statue or 
Athens' mini-Stonehenge. The secret spots category differentiates this application from competitors due 
to its focus on local attractions. 

The fourth task was to find a restaurant where a friend had recently been. This was accomplished by 
clicking on the users list of friends and selecting a friend. On that friends page you see where they have 
recently been. We did this again to illustrate that we can accomplish the same tasks as our competitor 
and provide information via a virtual form of "word-of-mouth." A participant in an evaluation was 
quoted saying "[Information] is gathered like word of mouth, but through the app." This task aimed to 
demonstrate to the participant how information could be gathered through friends and to discover if 
the feature successfully accomplished its goal. 

Task five was to find a different "Secret Spot". This was done in order to show the versatility of the 
application and that there are different ways of getting to the same page. You can search for things by 
tag and by going to the categories page and picking an attraction. 

We wanted to show that the application's content and interaction mimicked the "word of mouth" 
pattern we discovered in previous interviews and studies. The last two tasks had the user generate 
content of their own to explore whether or not the integration of virtual "word of mouth" was 
successful. The first of the two was to write a review. This allows the user to put in their personal 
thoughts on content. This is common practice in many applications. The ability to write a review gives 
the user a sense of contribution. And here, again, the application demonstrates that it has the features 
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one would expect from an application of this nature, in addition to the "Secret Spots" feature that sets 
this application apart. 

The last task was something that set us apart from some of the competitors. We asked the user to 
create a secret spot. Since the "word of mouth" pattern was so prevalent, this feature enables the user 
to be the first person to start the "word of mouth" chain. They can share any place that they feel is 
worthy and share it with the other users, again, virtualizing an interaction users would normally share 
during an in-person engagement. 

Results Discussion 

Immediately after viewing the results, adding a "secret spot" using the Town Crier application proved 
problematic for the participants. Of the eight subjects that participated in the think-aloud evaluation 
and retrospective test, five reported that they had difficulties while trying to add a secret spot. These 
five participants said that the button to add a spot was not immediately clear to them and was not in 
the pages they expected them to be in. One participant noted that the "+" icon was much too 
ambiguous on the front page and seemed like it was meant to add a status update or something similar 
- they did not expect it to add a venue. Another major reason for the confusion was the button's 
location. Participants noted that the button should be in multiple pages, not just the home page. Of the 
five participants that had difficulty adding a secret spot, three said that they expected the feature to be 
in the "Categories" page, among the list of venues and categories. 



Figure 3. User Performance Benchmarks - Adding a secret spot. 



Participant 


Time Elapsed 


Error Pages 


Total Pages 


A 


76 seconds 


4 pages 


5 pages 


B 


61 seconds 


3 pages 


4 pages 


C 


26 seconds 


4 pages 


5 pages 


D 


63 seconds 


5 pages 


6 pages 


E 


58 seconds 


3 pages 


4 pages 


Average 


57 seconds 


4 pages 


5 pages 



The participant's difficulty in adding a secret spot was further proven when looking at the performance 
measures in the usability testing session. The average time elapsed while a participant added a secret 
spot was 57 seconds, which would be higher if the outlier time of 26 seconds is ignored. In addition, the 
average number of pages visited to add a secret spot was five pages. For an experienced user, such as 
the conductor, the task is completed much quicker - only one page would need to be visited and as 
such, the time elapsed to complete the task is much shorter. The increased number of pages the 
participants visited occurred due to the lack of salient cues that the "add spot" button was located on 
the front page. Participants scoured other sections of the application to find a button that was located 
on one of the first pages they encountered. 

Given that the "secret spots" concept is the key feature of the application, the participant's experience 
with this feature would be of utmost importance if work continued on the application's development. 
To improve this feature in a future iteration, the "Add a spot" feature would be more clearly advertised 
and marked. The button would be made more prevalent - instead of being a simple "+" button, the label 
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would be changed to "Add Spot." The conductors of the heuristic evaluation also commented on the 
"Add Spot" feature, noting that it was difficult to find at first. As suggested by the heuristic evaluation, 
this feature can also be made more accessible by adding it to other pages of the application, such as the 
"Categories" page, or by adding it to the bottom navigation bar of the application. Participants did note 
that once the button was found, it was very intuitive to add a spot, therefore no other changes were 
proposed concerning adding secret spots. Users found it straightforward and simple to use the "Add 
Spot" feature after successfully inserting a spot into the application. 



Figure 4. Usability Testing and KSLM Comparison 



Task 


Usability Testing Time Elapsed 
(average in seconds) 


KSLM Time Elapsed 
(estimated in seconds) 


A 


41.4 


36.9 


B 


13.4 


5.8 


C 


24.2 


7.8 


D 


28.2 


3.9 


E 


6.0 


5.8 


F 


11.2 


8.0 


G 


57.0 


35.6 



KSLM provides estimates of how long it takes an expert user to complete a task. Certain tasks, such as 
adding a secret spot (Task G) and finding certain information (Task C and D) were prone to errors, 
resulting in poor performance compared to the Keystroke Level Model. 



Other minor issues within the application included inconsistency with the navigation bar. The heuristic 
evaluation, along with several participants from the think-aloud evaluation, noted that the navigation 
bar was not always present. In a future design, the navigation bar will be made available on every page 
for consistency. Results from the heuristic evaluation also noted that navigating to certain pages, such as 
category pages, had no back button and required the user to start from the home page. In a future 
design, back buttons will be included on every page accessed, further helping to make page navigation 
consistent. In addition, the heuristic evaluation found that there was no log out option - a feature that 
was not commented on by any participant. Resolving this issue would be very simple by including a log- 
out button in the "Settings" tab of the navigation bar. 

Issues aside, there were several concepts that the results of the evaluation proved successful. Aside 
from the page navigation issues noted above, participants noted that navigating the application was 
intuitive and simple - the ordering of the pages were logical and clear, the navigation bar was easy to 
use and provided the expected behavior, and the application as a whole was very organized. 

Figure 5. Elapsed time while finding restaurant information page. 



Participant 


Time Elapsed (Town Crier) in 
seconds 


Time Elapsed (Yelp) in seconds 


A 


18 


11 


B 


10 


25 


C 


13 


14 


D 


05 


10 


E 


21 


17 


Average 


13 


18 
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Figure 6. Number of total pages visited while finding restaurant information page. 
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On average, participants visited the same amount of pages while searching for a restaurant. 

Participants from the usability testing session noted that navigating the Town Crier application flowed 
well and closely matched the functionality of the third party application, Yelp. As seen in Figure 3, the 
average number of pages visited while participants searched for a restaurant's information page was the 
same - when looking for restaurant information, the Town Crier application was largely as efficient as 
using the popular application Yelp. Categories helped to quickly find venue information and were easily 
accessible. Participants disclosed that the Town Crier application distinguished itself from other 
applications by including the "Secret Spots" category as well as the feature of adding these "secret 
spots." Participants also noted that the Town Crier application provided search functionality that the 
Yelp application failed to provide. In the usability testing session, users were tasked to find all venues 
that were tagged as "Outdoor". The results would then show restaurants, venues, and attractions that 
had the tag. When given the same task on the Yelp platform, participants struggled to search for 
restaurants that were outdoors. Figures 4 and 5 show that the Town Crier application was more efficient 
to use when searching for venues by specific tag (i.e. outdoors). 

Figure 7. Elapsed time while searching for outdoor venues. 



Participant 


Time Elapsed (Town Crier) in 
seconds 


Time Elapsed (Yelp) in seconds 


A 


38 


76 


B 


16 


39 


C 


24 


56 


D 


14 


77 


E 


29 


43 


Average 


24 


58 
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Figure 8. Number of total pages while searching for outdoor venues. 
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In this respect, the Town Crier application succeeded in not only providing the functionality expected 
from an application that distributes venue information, it also succeeded in differentiating itself from 
competing applications. Participants of the think-aloud evaluation noted that adding secret spots made 
it simple to share information about a hole-in-the-wall venue to friends. They also commented as a 
whole, the feature would encourage users, primarily students, to venture into Athens and visit venues 
one would only find in the Classic City. Combining the "Secret Spots" feature with the ability to see what 
friends are visiting further encourages the experience, a participant said, by making it a way of spreading 
information by word of mouth while using the application. 

Figure 9. Participant responses when guestioned if they would use the "Add Spot" feature. 

Participant use of "Add Spot" Feature 




■ Yes, I would use it. ■ No, I would not use it. 
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All participants interviewed during the think-aloud evaluation noted that a friend leaving a good review 
of a venue or creating a new "secret spot" would heavily affect their decision on whether or not they 
would visit that venue. It is this combination, participants noted, that the application made itself unique 
from competitors, such as Yelp. It should be noted that two participants in the think-aloud evaluation 
noted that they would NOT use the "Add Spot" or "Add Comment" feature - they only wanted to use 
the application for discovering information, rather than contributing information. As seen in the post- 
evaluation survey responses, only a minority of participants (4 out of 15), would not use the "Add Spot" 
feature. 



Conclusion 




1^ 



Figure 10. Screenshot of application main page. 

The plus icon in the top right activates the "Add Spot" 
feature. The icon was noted to be confusing and did 
not provide salient cues for users searching for the 
"Add Spot" feature. 

In short, the Town Crier application did well to 
differentiate itself from standard applications while 
providing basic functionality expected from an 
application in this domain. It was as efficient as a 
major competitor when searching for venue 
information and was more efficient when searching 
for venues by a specific tag or keyword. It also 
provided a unique way to contribute and discover 
"secret spots" via a mobile application. This feature, 
combined with the application's feature of showing 
what friends have done, was noted by participants to 
be the factor that differentiates this application from 
its major competitor. Yelp. 

What the application failed to do was give the user a 
clear way to utilize the feature that made the 
application unique. In a future iteration, improving the 
unique "Add a spot" feature would give users an easy 
and much more intuitive way of using the application 
to spread information to their friends. 

Instead of a simple icon, the proposed change would be the addition of add an "Add Spot" button. This 
change, alongside the addition of a navigation bar on every page, addresses the major concerns from 
the participants of the retrospective and think-aloud evaluation. To address issues found in the heuristic 
evaluation (and have not been addressed during the other evaluations), a "log out" button will be added 
to the settings pages. To maintain consistency and standards, the button to go back from a page will be 
kept consistent across all screens - instead of having one page have a "back" button and another have a 
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simple arrow, all pages would have a textual "back" button. The heuristic evaluation also exposed minor 
bugs and graphical features that would be corrected, such as misaligned graphical features and non- 
responding buttons. 

Materials 

The following materials were used in the evaluations conducted on the Town Crier mobile application. 

Document 1. Consent Form 

Thank you for participating in this usability test for the Town Crier mobile application. During this 
test you will: 

• Be asked to perform a certain number of tasks 

• Asked questions about what action you take and why you take them 

• Be video recorded 

This test is completely voluntary test. You may leave at any time. If you feel uncomfortable with 
anything during the test please inform the tester. This test is strictly for the improvement of the 
Town Crier Application. All information gathered by the test will be kept confidential. Your name 
and personal information will at no time be used. 



If you have any questions please ask a member of Town Crier. 

I have read and understood this form and had all my questions answered. 



Signature Date 



Document 2. Testing and Evaluation Script 

Action: Have application open and ready for use. 

Hello, . Thank you for participating in this session today. My name is 

, and I will be explaining activities that we are going to perform in today's 

session. 

Before we begin, I will explain what we are here to do. I am going to be reading through this 
script in order to make sure that everything is covered correctly. 

As you may know we have asked you here to test a new smartphone application. We are getting 
participants to use the application in order to test how it can best be utilized and that it is 
working as planned. 
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We would like to make you understand that you are not being tested. The application is being 
tested. You cannot make a mistake during this test. What you do during the test will only give us 
information on how to make it better. Anything you say will be viewed as constructive criticism 
and not taken personally. At any time, if you have any questions please don't hesitate to ask. 

If you feel uncomfortable in any way you can end the test at any time. 

At this time we would like your permission to record the audio and video during the test. The 
recordings will only be used in order to help us recall this session and to help make the 
application better. If could please sign the consent form provided. It allows us to have a record 
of your permission to record our interactions and that the recordings will only be used for the 
purposes of the project. 

Action: Have participant sign form allowing us to record their actions as well as outlining the 
risks that they are taking by participating in this test. 

During the test I and any team member present will be recording certain actions that you are 
taking. These will include button touches, swipes, text typed and so on. We will not be recording 
anything about your person or that doesn't directly involve manipulating the application. Any 
questions before we get started? 

At this time I have just a few quick questions about you and your habit with a smartphone. 
What is your age? 

What is your occupation? Does it involve regular use of a smartphone? 

How much time would you say that you spend on your smartphone in an average day? 

How often do you frequent local venues (i.e. restaurants, bars, events) in the Athens area? 

Ok, thank you. We are done with those questions. Do you have any questions before I show you 
the application? 

Action: Show the participant the application. 

I am going to ask you to perform a number of specific tasks on the Town Crier application and 
on a similar application. Yelp. Please think about the differences between the two applications. 
The first task is to create a profile for the account. 

Action: Have the user create an account 

Thank You. Now that you have created a profile you should see the home page. I would like you 
to take a few moments to familiarize yourself with the page. You may scroll up or down but 
please don't touch any buttons yet. Once complete, create a profile on Yelp. You do not have to 
submit your login information. 

What is your first impression of the applications look? 

Can you visualize where different links on the page might lead you? 

Is there any icon or feature that you do not understand? 
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I will now ask that you try to find a few things. Please find the restaurant IVlama's Boy and tell 
me any information you find out about the restaurant. First, we will use the Town Crier app. 
Once complete, we will use a third party app "Yelp" to do a similar action. 

Action: Have the user find information about Mama's Boy. First use the Town Crier application. 
Once complete, do the same task on Yelp. 

Thank you. Now could you please search for all secret spots that have the tag "Outdoors". Tell 
me the attractions listed. Again, find the secret spots category and filter the tags by "Outdoors." 
Once complete, please attempt to find outdoor attractions through Yelp. 

Action: Have the user find all the secret spots that are tagged with "Outdoors" 

Thank you. Now using the Town Crier application, find friends and restaurants or venues they 
visited recently. Tell me the restaurant that they visited most recently. 

Action: Have the user find the restaurant that their friend most recently visited. 

Thank you. Please return to the "secret spots" category and find the name of the attraction. Tell 
me the name of the attraction. 

Action: Have the user find a secret spot. 

Thank you. Now could you please write a review of the secret spot you found in the previous 
task? The content of your review does not matter we are only interested in the process of 
posting the review. The review will be deleted after this session. 

Action: Have the user write a review for that secret spot (found above). 

Thank you. Now please think of your favorite park and add it as an attraction? Again, the place 
itself does not matter we are only interested in the process of adding an attraction. The 
attraction will be deleted after this session. 

Thank you. This concludes our testing. Do you have any questions, now that we are done? 
You have been very helpful. Again, thank you for your time your input has been very valuable. 

Document 3. Task List 

1. The first task is to create a profile for the account. 

2. Now that you have created a profile you should see the home page. I would like you to take a 
few moments to familiarize yourself with the page. 

3. Please find the restaurant Mama's Boy. 

4. Now could you please search for all secret spots that have the tag "Outdoors?" 

5. Now could you please find a friend in your list of friends? What restaurants/venues have they 
visited recently? 

6. Now please find a different secret spot and its description. 
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7. Now could you please write a review of the secret spot you found in the previous task? The 
content of your review does not matter we are only interested in the process of posting the 
review. The review will be deleted after this session. 

8. Now please think of your favorite park and add it as an attraction. Again, the place itself does 
not matter we are only interested in the process of adding an attraction. The attraction will be 
deleted after this session. 

Document 4. Questionnaire 



Town Crier review 

Age 



Gender 

Male 
k_i Female 

Are you a student? 

;> Yes 
No 

How often do you go out a week? 

0-2 times 
'Jj 3-5 times 
;j 6-7 times 
Kj more than 7 times 

How likely would you be to use a mobile application to find something new in Athens? 

12 3 4 
Very unlikely O O O O Very likely 

Please rank the following methods of finding out about a 
spot in Athens. 

1 being the easiest. 3 being the most difficult. 
By category. 



Through friends' activities. 
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How likely would you be to use a mobile application to find something new in Athens? 



Please rank the following methods of finding out about a 
spot in Athens. 



1 being the easiest. 3 being the most difficult. 
By category. 

Through friends' activities. 
By tag. 

Would you add a secret spot if you had an enjoyable experience at a location? 



No 

Overall, how was your experience using the Town Crier mobile application? 



12 3 4 



Very unlikely 



Very likely 



Yes 



/. 
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